Communication savings with ghost cell expansion for domain decompositions of finite difference grids

We investigate trade-offs between communications frequency and bandwidth, using the ghost cell expansion method (GCE) with domain decomposition for parallel evaluation of finite difference schemes for PDEs. GCE delays communication of boundary values between local domains for a small number of timesteps. This delay reduces communication pressure and aggregates messages into larger packets, thus optimizing for the high latency, high bandwidth interconnects common in modern parallel computing platforms. However, choosing the optimal ghost cell expansion level requires a careful balance between the cost of local computation per timestep, and network bandwidth (which may limit the effectiveness of increasing message size). Furthermore, for stencils wider than one cell on each side, GCE is an approximation which may impair numerical accuracy. For now, we neglect numerical aspects of GCE. We use a LogP model with bandwidth to show that for the platforms tested, the technique only has potential for reducing the amortized cost of communication in the 1-D and 2-D cases. The model suggests a declining payoff for increasing expansion level in the 2-D case. Experimental results with a test problem verify some of these predictions. 1 Ghost cell expansion Ghost cell expansion, proposed by Ding and He [6], is a form of domain decomposition with overlapping boundaries. Ding and He formulate it for finite difference methods used to solve PDEs with a timestep iteration. It accomodates either time-dependent PDEs (parabolic or hyperbolic) or elliptic problems E-Mail: [email protected] E-Mail: [email protected]